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Application Papers 
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* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application). 
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DETAILED ACTION 

Notice of Change in Art Unit 
1 The Group and/or Art Unit location of your application in the USPTO has changed. To 
aid in correlating any papers for this application, all further correspondence regarding this 
application should be directed to Group Art Unit 2671 . 

Allowable Subject Matter 
2. Claims 1-54, 56-63, 65-116, 118-129, 131-165, 237 and 240-243 are allowed. 
3 The following is a statement of reasons for the indication of allowable subject matter: The 
rendering of a schematic of the positions of the objects of the scene when the selected viewing 
direction is not within the predetermined range of viewing directions (claims 1 16, 129, 130, 177, 
203, 241-242 and 246); comparing the height of the representation of each object generated in 
dependence upon image data and generating object representations in the 3D computer model in 
dependence upon the height comparisons (claims 1,21, 23, 40 and 237); defining a model of the 
object in the 3D computer model in dependence on a determined ground footprint (claims 67, 84, 
85, 99 and 240); a second camera processing image data from a first camera, determining which 
image data is an object and which data is the object's shadow (claims 54, 63); defining a model of 
the object in the 3D computer model in dependence upon identified planar surfaces (claims 133, 
149, 150, 163, 243); and applying a transformation to the identified image data from a camera 
which defines a mapping from the ground plane in the space of the image data of the camera to a 
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surface in a modeling space (claims 44, 56, 238). 

4. Claims 2-20 and 42-43 are allowed because they depend, directly or indirectly, on claim 1 ; 
claims 24-39 are allowed because they depend, directly or indirectly, on claim 23; claims 45-53 
and 65-66 are allowed because they depend, directly or indirectly, on claim 44; claims 57-62 are 
allowed because they depend, directly or indirectly, on claim 56; claim 68-83 and 100-101 are 
allowed because they depend, directly or indirectly, on claim 67; claims 86-98 are allowed 
because they depend, directly or indirectly, on claim 85; claims 103-1 15 and 13 1-132 are allowed 
because they depend, directly or indirectly, on claim 102; claim 1 19-128 because they depend, 
directly or indirectly, on claim 1 18; claims 134-148 and 164-165 are allowed because they 
depend, directly or indirectly, on claim 133; claims 151-162 are allowed because they depend, 
directly or indirectly, on claim 150. 

5. Claims 16, 167-169, 180, 196, 208 and 216-217 are objected to as being dependent upon 
a rejected base claim, but would be allowable if rewritten in independent form including all of the 
limitations of the base claim and any intervening claims. 

6. The following is a statement of reasons for the indication of allowable subject matter: No 
prior art anticipates or suggests adjusting the effective light value of a light source in a virtual set 
patterned after a physical set to account for attenuation of each light source due to an angle 
between an optical axis of each light source and a line between each light source and the 
computer animated object (claim 16); generating information indicating the reliability of the image 
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data in dependence upon the angle between the user-selected viewing direction and the viewing 
direction upon which the image data was recorded (claims 167 and 180); consideration of the 
camera characteristics affecting quality in a predetermined order and values for each respective 
camera characteristic compared, with the determination of the representations to be rendered 
being made once the tests identify a characteristic which differs by more than a predetermined 
amount for given cameras (claims 196 and 208); image data for an object in a second image in a 
sequence is generated by rendering texture data based on image data from the first camera onto 
the first representation of the object in accordance with the second user-selected viewing direction 
(claims 216 and 228). Claims 168-169 and 181-182 are allowable because they depend on claim 
167 and 180, respectively. Claims 217 and 229 are allowable because they depend on allowable 
claims 216 and 228. 

Claim Rejections - 35 USC§103 

7. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 

rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art 
are such that the subject matter as a whole would have been obvious at the time the invention was made to 
a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be 
negatived by the manner in which the invention was made. 

8. Claim 166, 170-173, 175, 177, 179, 183-187, 189-195, 197, 199, 201, 203-207, 209, 
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212-213 and 244-246 are rejected under 35 U.S.C. 103(a) as being unpatentable over Marks 
(U.S. Pat. No. 6,025,847) in view of Diner et al. ("Diner," U.S. Pat. No. 5,182,641) and 
Goldberg et al. ("Goldberg," U.S. Pat. No. 5,877,779). 

9. With respect to claims 166, 177, 179, 187, 193-194 and 205-206, Marks, in disclosing a 
3D modeling system with visual feedback, also discloses a method of processing image data 
defining a sequence of images of at least one object moving in a scene to produce signals defining 
a representation of each object in a 3D computer model, and to generate image data by rendering 
an image of the 3D computer model, the method comprising processing the image data to identify 
image data relating to respective objects in the scene (Abstract, fourth and fifth sentences) and 
^fining a representation of each object in the 3D computer model in dependence upon the 
identified image data (Abstract, fourth and fifth sentences). 

10. However, Marks does not disclose generating image data by rendering an image of the 3D 
computer model in accordance with a user-selected viewing direction or quality information. The 
generation of image data is disclosed by claim 18 of the Diner composite video and graphics 
display for camera viewing systems in robotics and teleoperation, col.12, 11.33-57, and the quality 
information is disclosed by Diner at col.l, 11.31-41. 

1 1 Therefore, it would have been obvious to one of ordinary skill in the art at the time this 
invention was made to use the Diner system for real-time video display in the Marks modeling 
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system. This would save time for the user of the system by eliminating any need to move the 
cameras again once they have been initially set to view all locations of the workspace. 
12. However, neither Marks nor Diner disclose rendering texture data based on the identified 
image data onto the object representations. This is disclosed by the Goldberg method and 
apparatus for efficient rendering of 3D scenes at col.2, 11.5-1 1. 

13 Therefore, it would have been obvious to one of ordinary skill in the art at the time this 
invention was made to incorporate the Goldberg rendering method in the Marks modeling system. 
This would greatly increase the efficiency and throughput of graphics data in the rendering 
pipeline. 

14. The remainder of the claims in this rejection will now be considered: With respect to 
claims 170 and 183, Diner discloses more than one camera in col.l, 11.14-23, Marks discloses 
identifying image data from a camera relating to respective objects in the scene and defining a 
representation of each object in dependence upon the identified image data from the camera at 
col.6, 11.42-48, and Goldberg discloses rendering texture data based on the identified image data 
from at least one camera onto the object representations in col.2, 11.5-1 1 . 

1 5. Concerning claims 171 and 184, Marks discloses each object represented as a planar 
surface (implied by col.6, 11.39-41). 
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1 6. Regarding claims 1 72 and 1 85, Diner discloses quality information generated as pixel data 
within the generated image data (implied by col. 1, 11.3 1-41; it is well known that color is 
expressed in video graphics systems in pixels). 

17. With respect to claims 173 and 197, Diner discloses generating a signal conveying the 
image data and the quality information (also implied by col. 1, 11.3 1-41 ; it is well known that a 
signal needs to be generated in order to initiate and sustain a video graphics system). 

18. Concerning claims 175, 186, 199 and 209, Marks discloses displaying an image using the 
generated image data and displaying the quality information in the third sentence of the Abstract. 

19. Regarding claims 189 and 212, Marks discloses a storage medium storing instructions for 
causing a programmable processing apparatus to perform a method (memory device 104, FIG.l). 

20. With respect to claims 190 and 213, Marks discloses a signal conveying instructions for 
causing a programmable processing apparatus to perform a method (implied by computer 100, 
FIG.l). 

2 1 Concerning claims 1 9 1 , 203 and 246, Diner discloses the representation of each object 
rendered is determined in dependence upon the user-selected viewing direction, the respective 
viewing directions of cameras, and at least one camera characteristic affecting image data quality 
at col l, 11.31-41. 
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22. Regarding claims 192 and 204, Diner discloses the representation of each object rendered 
as being dependent upon the user-selected viewing direction, the viewing direction of respective 
cameras, and the resolution of respective cameras (col. 14, 11.24-30). 

23 . With respect to claims 1 95 and 207, Diner discloses a plurality of camera characteristics 
affecting image quality considered to determine the representation of each object for rendering at 
col.l, 11.31-41 (color and brightness). 

24. Concerning claim 20 1 , Goldberg discloses the selection of a representation of each object 
at col.l, 1.66 to col.2, 1.1. 

25. Therefore, in view of the foregoing, claims 166, 170-173, 175, 177, 179, 183-187, 189- 
195, 197, 199, 201, 203-207, 209, 212-213 and 244-246 are rejected as being unpatentable under 
35 U.S.C. 103 by Marks, Diner and Goldberg. 

26. Claim 174, 176, 198 and 200 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Marks, Diner and Goldberg and further in view of the Sharp Corporation's Model VC- 
5W20E Video Cassette Recorder ("the Sharp VCR"). 

27. Neither Marks, Diner nor Goldberg disclose recording the signal conveying the image data 
and the quality information. However, this is disclosed by the Sharp VCR at page 15. 

28. Therefore, it would have been obvious to one of ordinary skill in the art at the time this 
invention was made to incorporate a VCR into the Marks-Diner-Goldberg 3D modeling system. 
This would preserve the image data for later use. 
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29. Accordingly, in view of the foregoing, claims 174, 176, 198 and 200 are rejected as being 
unpatentable under 35 U.S.C. 103 by Marks, Diner, Goldberg and the Sharp VCR. 

30. Claims 178 and 188 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Marks and Diner according to the same rationale as claim 166; the quality information is implicitly 
indicated by its presence in a way similar to the way it is indicated at col. 1, 11.3 1-41 of Diner. 

3 1 . Claim 202 and 210-21 1 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Marks, Diner and Petruchik (U.S. Pat. No. 5,587,752). 

32. With respect to all three claims, neither Marks nor Diner disclose selecting image data to 
be used to define the object from a plurality of images of the object each recorded by a specific 
camera. This element is disclosed in the Petruchik camera, system and method for producing a 
composite photographic image at col. 10, 1.65-col.l 1, 1.2. 

33 Therefore, it would have been obvious to one of ordinary skill in the art at the time this 
invention was made to incorporate the Petruchik method into the Marks-Diner 3D modeling 
system. This would provide improved focus and exposure balance on an object. 

34. Accordingly, in view of the foregoing, claims 202 and 2 1 0-2 1 1 are rejected as being 
unpatentable under 35 U.S.C. 103 by Marks, Diner and Petruchik. 

35. Claims 214-215, 218-219, 221, 223-227, 230-236, and 247-248 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Marks, Diner, Goldberg and Petruchik and further in 
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view of applicant's admitted prior art Haralick et al., Computer a nd Robot Vision Volume 2 
("Haralick"), and Wolberg, Digital Image Warping . 

36. With respect to claims 214, 218, 223-226, 232-234 and 247-248, neither Marks, Diner, 
Goldberg nor Petruchik disclose testing whether first and second images of the object displayed 
from the generated image data will be discontinuous by testing whether the image data for the 
object in the second image in the sequence differs by more than a predetermined amount from 
predetermined image data. But according to p.49, 11.3-6 of this application's specification, which 
refers to p.48, 11. 12-19, this element is disclosed by Haralick. 

37. Therefore, it would have been obvious to one of ordinary skill in the art at the time this 
invention was made to incorporate the Haralick method into the Marks-Diner-Goldberg-Petrucik 
3D modeling system. This would facilitate the diagnosis of the problem of discontinuous images 
by enabling comparison of the images to find out how similar they are. 

38. Neither Marks, Diner, Goldberg, Petruchik nor Haralick disclose generating modified 
image data for the object in the second image if the image data for the object in the second image 
differs by more than the predetermined amount. But according to p.49, 11.8-17 of this 
application's specification, this element is disclosed by Wolberg. 

39. Therefore, it would have been obvious to one of ordinary skill in the art at the time this 
invention was made to incorporate the Wolberg method into the Marks-Diner-Goldberg-Petrucik- 
Haralick 3D modeling system. This would enhance clarity of the image by eliminating 
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discontinuity in the image. 

40. The rest of the claims in this rejection will now be considered: With respect to claims 215 
and 227, Goldberg discloses rendering a representation of an object at col.l, 1.66 to col.2, 1.1. 

41 . Claims 219, 221 and 23 1 are rejected with the same rationale as claim 173. Claim 235 is 
rejected with the same rationale as claim 189. Claim 236 is rejected with the same rationale as 
claim 190. 

42. Therefore, in view of the foregoing, claims 214-215, 218-219, 221, 223-227, 230-236 and 
247-248 are rejected as being unpatentable under 35 U.S.C. 103 by Marks, Diner, Goldberg, 
Petruchik, Haralick and Wolberg. 

43 . Claims 220 and 222 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Marks, Diner, Goldberg, Petruchik, Haralick and Wolberg and further in view of the Sharp VCR 
according to the same rationale used to reject claim 175. 

Claim Rejections - 35 USC§102 

44. The following is a quotation of 35 U.S.C. 102(b) which forms the basis for all novelty- 
related rejections set forth in this Office action: 

A person shall be entitled to a patent unless — 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or 
sale in this country, more than one year prior to the date of application for patent in the United States. 

45. Claims 178 and 188 are rejected under 35 U.S.C. 102(b) as being anticipated by Diner. 
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46. With respect to both claims, Diner discloses generating image data by rendering an image 
of the 3D computer model in accordance with a user-selected viewing direction or quality 
information. The generation of image data is disclosed by claim 18 of the Diner composite video 
and graphics display for camera viewing systems in robotics and teleoperation, col. 12, 11.33-57, 
and the quality information is disclosed by Diner at col. 1, 11.3 1-41. 

47. Therefore, in view of the foregoing, claims 178 and 188 are rejected as being anticipated 
by35U.S.C. 102(b) by Diner. 

48. The following is a quotation of 35 U.S.C. 102(e) which forms the basis for all novelty- 
related rejections set forth in this Office action: 

A person shall be entitled to a patent unless — 

(e) the invention was described in a patent granted on an application for patent by another filed in the United States 
before the invention thereof by the applicant for patent, or on an international application by another who has 
fulfilled the requirements of paragraphs (1), (2), and (4) of section 371(c) of this title before the invention thereof 
by applicant for patent. 

49. Claims 55 and 239 are rejected under 35 U.S.C. 102(e) as being anticipated by Tsao (U.S. 
Pat. No. 5,911,035). 

50. Tsao, in disclosing a method and apparatus for determining binocular affine disparity and 
affine invariant distance between two image patterns, also discloses, with respect to both claims, a 
method of generating a model of an object in a 3D computer model in which a transformation is 
applied to image data from a camera relating to the object and its shadow which maps the image 
data for one of the object and its shadow to a surface (col.3, 11.8-15. The transformation is 
applied to image data relating to object mapping image data. Since col. 8, 11.41-63 discloses two 
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cameras, it would be obvious to apply this same transformation to data from a second camera.) 
Tsao also discloses modelling the object in dependence upon part of the transformed image data 
at col.2, 11.63-66. 

5 1 . Therefore, in view of the foregoing, claims 55 and 239 are rejected as being anticipated 
under 35 U.S.C. 102(e) by Tsao. 

Conclusion 

52. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Lance Sealey whose telephone number is (703) 305-0026. The 
examiner can normally be reached Monday-Friday from 7:00 am to 3:30 pm EDT. 

53 . If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mark Zimmerman, can be reached on (703) 305-9798. The fax phone number for the 
organization where this application or proceeding is assigned is (703) 872-93 14. 

54. Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305-4700 or 
the Customer Service Office at (703) 306-0377. ^ 




MARK ZIMMERMAN 
SUPERVISORY PATENT EXAMINER 
TECHNOLOGY CENTER 2600 



